Retry retryable ack failures instead of abandoning the work item (redux) - #72

Summary

Fixes #41. Taking over for #42.

The worker dispatcher previously abandoned a work item on every acknowledgement error, including retryable provider failures. Abandonment discards the completed activity's result and redelivers the item, potentially running the handler again. That is permitted by at-least-once execution, but is avoidable for transient acknowledgement failures.

This change retries acknowledgements in place for activity success, application errors, and cancellation. There are at most five total attempts, with waits of 100, 200, 400, and 800 milliseconds between attempts. The waits total 1.5 seconds of cumulative backoff, excluding provider-call duration; this is not a wall-clock deadline. Permanent errors and exhausted retries use the existing abandonment path.

The rebased worker retains lock renewal while a normal completion's acknowledgement is pending. Activities still require at-least-once-safe behavior: crashes, lost locks, and exhausted retries can cause re-execution. Poison-message handling, provider APIs, persisted schemas, and SQLite pool configuration are unchanged.

The tests now check exact acknowledgement/abandonment sequences, lock tokens, and completion payloads:

• Success on the fifth and final attempt without re-executing the handler.
• Five failed attempts, then abandonment and successful redelivery.
• A permanent error followed immediately by abandonment, not an in-place retry.
• Retried application-error completions without re-executing the handler.
• Retried cancellation acknowledgements with no completion payload.
• An acknowledgement that commits but reports a retryable error: the retry encounters the missing token, stops, and does not duplicate the activity or its completion.

These fault-injection tests use file-backed SQLite so shared-cache deadlocks do not interfere with their exact call-count assertions. The existing in-memory, multi-worker session test remains a separate regression check.

Checklist

• Tests added/updated
• Regular tests pass locally (cargo nt)
• Doc tests pass locally (cargo test --doc --all-features)
• Docs updated
  • Existing docs updated where behavior changed
  • New doc added under docs/ if introducing a new area
  • Linked from docs/README.md

Links / Design notes

SQLite diagnosis

The original issue and PR description attributed the shared-cache failure to SQLx not using SQLite unlock notifications. That explanation is incorrect for SQLx 0.8.6, used in this checkout: its statement execution loop already calls sqlite3_unlock_notify, and the dependency enables unlock-notification support.

SQLite emits the reported database is deadlocked error when unlock-notification registration detects a wait cycle. Waiting longer cannot resolve that cycle. SQLite documents rollback and later retry of the transaction as the recovery mechanism. The failed acknowledgement transaction is rolled back,
and the next provider call opens a fresh transaction.

This supports retrying the complete acknowledgement operation rather than rerunning the activity. The same policy also applies to other providers' retryable failures. A connection failure may leave a commit outcome unknown; atomic deletion plus lock-token validation prevents a repeated acknowledgement from enqueueing the completion again if the first call already committed.

Historical measurements

With the strict, exactly-once assertion retained as a measurement probe, Pino reported 3 failures in 14 runs before the runtime fix and 0 in 20 after it. The latter runs recorded 19 acknowledgement retries and 741 SQLITE_LOCKED events. Those measurements were not repeated during the takeover; the current validation uses deterministic fault injection and the regression tests listed above.